47. GPT-2 Architecture
GPT-2 is a decoder-only Transformer language model.
Its architecture contains:
- token embeddings
- positional embeddings
- Transformer blocks
- masked self-attention
- feed-forward networks
- residual connections
- layer normalization
- language-model output
Understanding the architecture makes it possible to reproduce the model structure and load compatible pretrained weights.